Online Knowledge Distillation via Multi-branch Diversity Enhancement

Knowledge distillation is an effective method to transfer the knowledge from cumbersome teacher model lightweight student model. Online uses ensembled prediction results of multiple models as soft targets train each However, homogenization problem will lead difficulty in further improving performance. In this work, we propose a new enhance diversity among models. We introduce Feature Fusion Module (FFM), which improves performance attention mechanism network by integrating rich semantic information contained last block Furthermore, use Classifier Diversification(CD) loss function strengthen differences between and deliver better ensemble result. Extensive experiments proved that our significantly enhances brings evaluate on three image classification datasets: CIFAR-10/100 CINIC-10. The show achieves state-of-the-art these datasets.